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ABSTRACT 

In the f/rst report, dated August 1991, assessment of 
three language tests is reported. The tests were developed and are 
used as part of a Los Angeles Unified School District (California) 
program to teach mainstream American English (MAE) to speakers of 
African American Language (AAL) . The tests include a writing 
assessment (grades 3-6) and two instruments for evaluating oral 
skills (grades K'^6) , all designed to help assess program 
effectiveness. Test development, including pilot testing and 
revision, is described and conclusions and recommendations are 
presented. Appended materials include directions for written test 
administration, writing test topics, student writing samples (with 
scoring), writing assessment forms, notes on pretest error patterns, 
directions for oral test administration (both structured task and 
free production), scoring sheets, and sample responses (transcribed). 
The second report, dated December 1992, describes test modifications 
made in the project's second year. First, changes in the two 
prototype speaking tests are discussed, including development of a 
new prompting format for the free production test and pilot testing 
and revision processes and results. Creation of new writing test 
prompts is then described; Appended materials include examples of the 
10 features of difference between MAE and AAL being tested, test 
administration directions, forms, and visual prompts* (MSE) 
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Introduction^ 



The Los Angeles Unified School District (LAUSD) has been developing a 
special language arts curricultxm to provide teachers with techniques and 
resources for teaching Mainstream American English (MAE) to speakers of 
African American Language (AAL). A primary goal of the program is to help 
African American students develop proficiency in the use of oral and written 
MAE so that they will have a choice of language alternatives. Developing this 
proficiency should thereby increase the academic and future life success of 
these African American students. 

The UCLA Center for the Study of Evaluation has developed three 
prototype evaluation instruments^ — one for evaluating writing (grades 3-6), and 
two for evaluating speaking (grades K-6) — to help the LAUSD assess the 
program's effects on student proficiency in MAE. This paper reports on the 
development of the prototype instruments for assessing written and oral MAE. 
Both the test development effort reported here and the prototype instruments 
should be viewed as preliminary. Additional refinement of test format and 
subsequent validation are needed before the instruments can be used for 
evaluation purposes. 

This paper is organized as follows: The development of the writing test is 
discussed first, including the pilot testing procedure and results. Conclusions 
and recommendations are provided. Next, the development of the speaking 
tests is presented, including pilot testing procedures, results, subsequent 
revisions in format and approach, conclusions, and recommendations. 
Finally there is a summary section. 

Assessment Instruments 

The two types of instruments for assessing the effectiveness of the English 
language curriculum for African American students are discussed in turn 
below. In preparation for developing the instruments, relevant literature was 



l The authors wish to thank Dr. Lorraine Cole, Director, Office of Minority 
Concerns, American Speech-Language-Hearing Association, for her technical 
assistance in addressing the issues involved in this test development project. 



reviewed and existing language tests were examined. Our approach to the 
testing effort grew out of the literature review and review of program goals and 
curriculimi plans. These led us to focus on the differences between AAL and 
MAE. Indeed, both the writing and oral instruments focus exclusively on 
differences between the two varieties of English. The content base for the 
instruments is a LAUSD document entitled ^^Characteristic Linguistic 
Features of African American Language Contrasted with Linguistic features 
of Mainstream American English.'* 

Writing Test 

The writing test involves obtaining a sample of each student's writing and 
evaluating the sample using the Writing Assessment Scoring Scheme 
developed by the Center for the Study of Evaluation (CSE). Obtaining the 
writing sample is a two-day process. The rationale for a process approach is 
that by allowing students ample time to write and revise, we will obtain 
writing samples that accurately reflect the students' abilities to write in MAE. 
Students are told they have two class periods, approximately 30 to 40 minutes 
each day, to write and revise their stories. They are asked to write the best 
stories they can using MAE. 

For this pilot testing effort, there were two main areas of inquiry. First we 
wanted to see whether or not the topic types and the specific topics we had 
selected would, in fact, elicit ratable samples of language from the students. A 
corollary to this was the issue of whether or not there would be variations in 
the effectiveness of the prompts by grade level. Second, we wanted to see to 
what extent the rating scheme we drafted could be used to evaluate the writing 
samples for MAE usage. 

Pilot testing for writing took place in grades 3-6 with two classes per grade 
level. Most of the testing was conducted at two schools in the district, Baldwin 
Hills School and Figueroa Street School. These two schools were selected on 
the basis of differing demographic features which included especially 
socioeconomic variables. However, since Figueroa Street School does not have 
sixth-grade classes, 99th Street School provided the second sixth-grade class. 
Table 1 provides the configuration for pilot testing the writing prompts. 
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Table 1 



Number of Students Tested per Grade pei School 
and Total Number of Students Tested per Grade 
for Writing 







Grade 




School 


3 


4 


6 


6 


Baldwin Hills 


23 


23 


25 


16 


Figueroa St. 


20 


21 


24 




99th Street 








19 


Total 


43 


44 


49 


35 



Thire was a relatively even distribution of students per grade level which 
was important. We wanted to know how well individual prompts worked in 
obtaining ratable samples from the students at different grade levels. 

Four potential writing topics were pretested, two with pictures and two 
without. The four topics were equally and randomly distributed in each class. 
We asked the teachers to circulate among the students and answer questions 
about specific topics. Directions for the test administrator are provided in 
Appendix A. The four topics, abbreviated in the tables as (a) special memory, 
(b) new friend, (c) boy and cat, and (d) girl and parents, are provided in 
Appendix B. Table 2 provides the initial distribution of students by grade, 
topic, and school. Table 3 provides the same information with incomplete 
(student absent the second day) and incoherent samples eliminated- Even 
after the unusable samples were eliminated, the distribution of topics 
remained good. We ended up with almost equal numbers of writing samples 
for each topic. 

All four topics elicited ratable samples of language from students of all 
grade levels tested. Indeed the students seemed to clearly understand the 
specific task prompted by the instructions for each topic. Variations in student 
responses were, however, apparent across grade levels and in response to 
topics which either did or did not include a picture in the instructions. 
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Across the four topics, students in grades 5 and 6 wrote stories which 
were generally longer and included more of the grammatical features 
assessed by the scoring scheme than students in grades 3 and 4. (See 
Appendix C for a set of four stories based on the same prompt, one each from 
grades 3-6.) With regard to the general coherence of student stories, those 
written in response to topics with pictures in the instructions were clearer 
than those written in response to topics which did not include a picture in the 
instructions. Topics presented with pictures also seemed to rouse greater 
creativity in student responses. (See Appendix D for two sets of two stories 
each — one set from grade 4 and one set from grade 5— which compare stories 
based on sentence prompts to stories based on picture prompts.) 

The variation in responses to topics is most evident in the written samples 
of third-grade students. It may be that the third-grade students as a group 
respond better or more easily to visual stimuli. For the yovmger students at 
least, picture prompts should probably be used. 

Two forms of the Writing Assessment Scoring Scheme are included in 
Appendix E. The first, the Tentative Scoring Scheme, is the research model 
used to evaluate the writing samples from the pilot testing effort; the second is 
the revised form modified on the basis of the pilot. The grammatical features 
on the scoring scheme are points of distinction between written MAE and AAL 
as delineated in the LAUSD document cited above. With the exception of the 
second person plural (i.e., you ^ you all), all features were observed in both 
MAE and AAL in student samples. Because the second person plural, ""you 
all,** tends to be acceptable colloquial usage in both varieties of English, we 
decided to delete it as a feature from the scoring scheme. 

While the original scoring scheme was thorough in the coverage of 
feature differences between MAE and AAL, it was time consuming to apply, 
requiring an average of five minutes per written page, and needed a rater 
familiar with the distinctive features of written MAE and AAL. In addition, it 
was difficult at times to determine the source of an error, that is, whether its 
occurrence was attributable to the student's use of MAE or AAL, a mechanical 
error, or the student^s normal writing development. This issue had an impact 
on scoring time. (See Appendix F for a list of problematic usage that occurred 
in the pretest samples.) In training raters to use the Writing Assessment 



Scoring Scheme, the identification of error types not readily attributable to 
differences between MAE and AAL must be covered. 

In addition to removing second person plural as a feature, we further 
modified the scoring scheme after the pretest samples were scored. Under the 
category "Grammatical Features," the actual feature marker was added to the 
box containing the name of the feature. For example, within the box for 
Pluralization, we added "plural marker -s.** This was done in turn for all of 
the features. We decided that more thoroughly specifying the features would 
facilitate the evaluation process. The section of the scoring scheme under 
Usage was modified to demonstrate the mathematical scoring process 
involved. The addition sign (+) and equals sign (=) were added to the scheme to 
show that the nimiber of occurrences identified as MAE for a particular 
feature are added to the number of occurrences identified as OTHER (AAL 
usage ar.d usage not clearly identifiable as either variety) for the same feature 
to yield the total number of occurrences for that feature in the writing sample. 
That is, MAE + OTHER = FEATURE TOTAL (see Appendix E). 

Besides adding the number of occurrences for each feature, the number of 
occurrences across all features for MAE «.*d for OTHER respectively are 
added, and the total written in the appropriate box for each next to Usage 
Grand Total. To obtain an index of MAE Usage for the sample, the MAE 
Usage Grand Total is divided by the Feature Grand Total. The index of MAE 
Usage for a given sample takes into account the proportion of occurrences of 
MAE to OTHER for a particular feature and also the total number of MAE 
features in proportion to the total number of occurrences of all the specified 
features in a writing sample. Thus the index of MAE Usage for a piece of 
writing will provide some indication of the student's control of MAE in free 
composition. A limitation, of course, is that certain topics will lend themselves 
to the use of some features more than others. Still, the proposed index of MAE 
should prove to be a viable means for judging pre/post performance vis-a-vis 
the English language curriculum for African American students. 

Conclusions. The two-day process approach to collecting writing samples 
worked very well with the students tested. For this reason, we suggest 
retaining this procedure for the test. 



The pilot testing restilts indicate that while all of the topics used elicited 
ratable samples of writing from the children, the two topics based on pictures 
elicited samples that were richer in overall language use and also provided 
more of the features being assessed. For this reason, we suggest using picture 
topics in the next stage of testing. Other picture prompts should be selected for 
further piloting so that two forms of the test will be available. In this way, one 
form can be used at the beginning of the school year and one at the end of the 
school year once the test is implemented. 

The Writing Assessment Scoring Scheme, while showing promise, 
requires further trials before we can feel confident about its use. Most 
problematic is the issue of identifying and categorizing feature occurrences 
that reflect neither MAE nor AAL usage. The revised draft of the scoring 
scheme (Appendix E) needs validation. 

Recommendations. Based on the pilot test results, we recommend that 
during the next state of writing test development: 

the writing sample be collected over two days as in the initial pilot; 



• additional topics with pictures be piloted to generate a second form of 
the test; 

• the revised Writing Assessment Scoring Scheme be used; 

• a training procedure be developed for raters that includes an overview 
of the scheme, examples of use, repe ed cycles of practice and 
feedback, and reliability checks; 

• in addition to a second stage of pilot testing, validation studies be 
conducted during the next academic year. 

Speaking Tests 

Two speaking tests were pilot tested: a test which was a structured task 
and one which was a free production task. Every student was asked to take 
both tests. Our plan was to obtain two samples of speech from each student 
that were different in kind. We hoped that by doing so we would be able to 
assess the student's use of MAE with some degree of confidence. Both tests 
were tape recorded for subsequent review. The two tests are discussed in turn 
below. 

15 



ERIC 



The pilot testing for speaking took place in grades K, 2, 4, and 6. As with 
the writing pilot study, most of the testing was conducted at Baldwin Hills 
School and Figueroa Street SchooL However, 99th Street School again provided 
sixth-grade students. Table 4 provides the configuration for testing the 
speaking tasks. 

Table 4 

Number of Students Tested per Class per Grade for 
Speaking 



Grade 



School 


Class 


K 


2 


4 


6 


Baldwin Hills 


A 


4 


4 


4 


4 




B 


4 


4 


4 


4 


Figueroa St 


C 


4 


4 


4 






D 


4 


4 


4 




99th Street 


E 








4 




P 








4 




Total 


16 


16 


16 


16 



In each school, eight students per grade level were selected from two classes. 
A total of 16 students per grade level were tested. A small nxmiber of students 
per class were tested so that the testing would not burden the teachers. The 
teachers w^ere asked to select the students to be tested, choosing as best they 
could a range of students in terms of both proficiency with MAE and general 
verbal ability. 

Structured test. The structured test involved having the test 
administrator/teacher read 10 sentences to each student and having the 
student repeat exactly what the administrator said. The structured test 
allowed us to use exactly the same measure with each student. Consequently, 
we were able to control the language environment and thus guarantee 
opportunities for producing specific language features. 



The structured test was administered individually to each student and 
scored on the spot by the test administrator. The test administrator used a 
scoring sheet with each student which contained the ten sentences. The 
specific language features being tested were underlined. (See Appendix G for 
the directions to the test administrator and the scoring sheet for the structured 
test.) Fourteen features of difference between MAE and AAL were tested. 
Some sentences contained a single feature while some contained more than 
one feature. Across from each sentence, the specific feature or features were 
listed with a blank next to tlie feature. The test administrator was instructed 
to not write anything in the blank if the student used MAE, If the student did 
not use MAE, the administrator was to check on the appropriate line. Table 5 
shows the score (nmnber of MAE productions) distribution across grade levels 
and classes for the 64 students who took the structured test. 

As mentioned above, there were fourteen features tested. If a student 
repeated all of the features in MAE, as many did, his/her score was 14. The 
overall mean on the test was 11.6 which indicates a high, though not complete, 
use of MAE for the features tested with the sample of students in the study. 

Of the fourteen features tested, four repeatedly varied in the students' 
speech from the sentences spoken by the test administrator in that the speech 
of the students conformed to the rules of AAL. There was variation with other 
features as well, but not to the same degree as with the four below: 



Feature and sentence 


Number out of 64 whose 
speech conformed to AAL 


• 


- indefinite article 

Jesse has an aunt v/ho lives next door. 


31 




- sound cluster (word final) 
The two pencils are on the desk* 


20 


• 


- reflexive pronoun 

The little bov hurt himself when he fell. 19 




- soimd cluster /Id/ 

Your brother callfji me yesterday. 


17 


• 



10 



17 
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Tables 



Score Distribution Across Grade Levels and Classes 
for the Structured Test 



Baldwin Hills 
School 

K K 



9^ 9 

11 32 

13 14 

14 14 

(X=12) 



Figueroa Street 
School 

K Ka 



8 5 

11 11 

12 11 

13 12 

(X=10.4) 



2 


2 


2 


2 


1« 


10 


10 


12 


9 


10 


13 


12 


12 


9 


10 


14 


14 


13 






14 


14 


14 







(X=12.6) (X=ll.l) 



4 4 



8 11 

10 11 

10 u 

12 12 

(X=10.6) 



4 4 



11 7 

11 10 

12 10 

13 12 

(X=10.8) 



99th Street School 
6 6 6 6 



11 11 

12 12 
12 14 
12 14 

(X=12.3) 



12 13 

14 14 

14 14 

14 14 

(X=13.6) 



* No free speech samples were obtained from this 
class. Students could not respond to prompts. 

^ Numbers indicate correct repetitions in MAE for 
the 14 features tested. 

This class included first and second graders, so the 
teacher selected two students from each grade level. 
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The structured task was understood by all of the students and was 
completed without difficulty. The task provided a range of performance on the 
fourteen features identified as differences between MAE and AAL. Whether 
variation from MAE in the repetition task was due to not having that feature as 
part of a productive repertoire is not clear from the data because each feature 
was limited to one occurrence. Lorraine Cole (personal communication) 
suggested that in revising the task, the number of features tested be limited to 
ten and each feature be assessed twice. Further, she suggested criteria for 
feature selection which guided our revision. 

The criteria identified by Cole as ones to consider for selecting features to 
test are: 

1. Frequency of occurrence 

2. Category of feature (non-obligatory/other) 

3. Developmental appropriateness 

4. Most versus least stigmatized 

5. What is most important? (Ask teachers, look at curricxilum) 

6. Skill level of evaluator (Eliminate features that are most problematic to 
analyze) 

Given the nature of the testing situation — children tested individually — 
and the need for both expedience and accuracy^ we decided to focus on 
Criterion 6 when revising the list of sentences and eliminate the features 
which in the pilot testing seemed consistently most difficxilt for an untrained 
rater to hear. Those features were the word final soimd clusters /sk/ and /Id/ 
as in the following three sentences taken from the test. 

The two pencils are on the defit. 

John s car is old. 

Your brother called me yesterday. 

We also decided to eliminate the stress feature tested in the sentence below 
because it was an issue with only two of the 64 students in the pilot test sample. 
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The xiolica are in the store. 
AAL: police MAE: police' 

Another consideration in our revision of the structured test had to do with 
the interest level of the sentence content for young students. At Cole's 
suggestion, we attempted to produce sentences that woxild be more interesting 
for the target population than the sentences that were pilot tested. Along these 
lines, it might be possible to further increase the interest level by 
accompanying the sentences with pictures. 

In AAL, certain linguistic markers do not exist, that is, plural markers, 
possessive markers, third person singular markers, past tense markers. 
Similar sound omissions can be due to articulation or conceptual problems 
and can be mistaken for AAL usage. To verify that an unmarked feature is 
truly indicative of AAL usage and not an indication of an articulation or 
conceptual problem. Cole recommended that a brief pretest be added to the 
structural test to rule out the latter. 

The pretest consists of having the test administrator ask the student to 
(a) repeat a list of words containing the soimds being tested, /s, z, iz/ at the end 
of the word, (b) show the student two pictures, and (c) ask a question about 
each picture that requires an answer that uses the full (uncontracted) copula. 
To the student, these tasks will appear to be part of the total test and will not 
actually be called a pretest. If the student cannot produce the features being 
checked by the words and/or does not produce the full form of the copula, the 
test administrator will note this on the scoring sheet and will either stop the 
test at the end of the pretest or continue only with a few sentences on the actual 
structured test so that the student will not be discouraged. 

The scoring sheet for the structured speaking test was revised based on 
the issues discussed above. It now consists of a pretest and 16 sentences with 
10 features, each one tested twice. Some sentences contain one feature and 
others contain two. A perfect score on the revised structured task is 20. The 
score can be reported as either a raw score of, for example, 18/20, or as a 
percent correct, in this case, 90%. The pretest is not scored. The revised 
scoring sheet with the pretest and new sentences is provided in Appendix H. 
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Careful attention must be given to training the test administrators to 
focus on natural delivery of the sentences to be repeated. In the pilot testing, 
the manner in which the sentences were read varied. One method to 
standardize the delivery of the sentences might be to tape record them and 
have them presented to the students on cassette. 

Administration and scoring time are important issues. In contrast to the 
writing task, each assessment must be individually administered, one-on-one. 
In the pilot test, each administration required about two minutes: This 
represents significant time to complete an entire class. 

Free production test For the free production test, each student was given 
a pi' ture and a verbal prompt and asked to tell a story in MAE based on the 
picture. The student was asked to tell his/her story to a small group of 
classmates (3-5). Each student had approximately one minute to look at the 
picture and think about what to say. Four pictures with short oral prompts 
were pilot tested. The directions to the test administrator, including the 
pictures and oral prompts, are provided in Appendix I. The draft scoring 
sheet for the free production test is included in Appendix J. 

With the exception of the four students in one kindergarten class at 
Figueroa Street School (see Table 5), all of the students seemed to understand 
the free production task and produced some language in response to the 
pictures they were given. As expected, there was an increase in the amount of 
language produced by the students by grade level. That is, in general, the 
second graders produced more language than the kindergartners, and so on; 
but there was considerable variation at each grade level. The task was 
especially difficult for the kindergartners. They had to be prompted quite a bit. 
One kindergarten teacher commented that the children seemed unable to 
commtmicate their thoughts; they were unable to fantasize. This implies that, 
for kindergarten students, a free production task such as this pilot test, is too 
difficult to elicit language samples representative of the students* language 
ability. 

Across all grade levels the types of responses varied considerably. Some 
students described the picture, some listed what they saw in the picture, some 
repeated the verbal prompt with slight variation, some produced short 
narratives about the person or persons in the picture, and still others used the 
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picture as a spin off for narratives only slightly related to the picture. 
Appendix K provides transcribed responses by grade level for each of the four 
prompts. 

There were some instances of non-MAE usage in the students' language, 
but those instances were few and there was no discemable pattern either in 
features or grade level. In all cases, the language samples were too brief to 
allow us to draw any conclusions with confidence. Cole suggests that 50 
utterances would be a minimally acceptable sample per student on a free 
production task. Samples from all grade levels were much shorter than 50 
utterances. Either the task or the age of the students, or both, mitigated 
against our obtaining adequate samples from the pilot testing with the free 
production test. 

Like the structured oral test> the free production test also raises problems 
of administration and scoring time, particularly for the longer samples 
required for reliable assessment. 

Conclusions. The proposed sentence repetition test elicits ratable speech 
samples from K-6 students by providing a controlled language environment in 
which feature differences between MAE and AAL can be examined. The 
structured nature of the task is both its strength and its weakness, however. 
The strength of the task is that it allows for specifically identifying testable 
features, and then provides the environment for those features to occur. The 
weakness of the task is precisely its restrictive nature. It does not allow for 
natural language use. We are not sure whether and how the picture of the 
students' language proficiency is distorted by not allowing for natural 
language use. But given the limitations of time and resources, the structured 
test appears to provide a better indication of a student's control of MAE than 
does the fi*ee production test. 

We have revised the scoring sheet for the structured test and expanded it 
to include a pretest. While the scoring sheet worked quite well in its initial 
form, the revised form must now be pilot tested. 

Since the free production task did not prove to be an effective means of 
obtaining a ratable sample of speech from the students tested, two possible 
alternative formats could be considered for additional piloting. One format 
would involve using a story of paragraph length loaded with the features to be 
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tested and having the students retell the story. Paragraph length can be varied 
with the grade level of the students. A second potential format involves having 
the test administrator tell a story and then ask structured questions about the 
story. Story length could vary by grade level. With both of these formats, the 
language environment is more controlled than in a free production task, 
allowing for focus on the feature differences between MAE and AAL. Time 
requirements, however, are a likely concern, and in this regard, validation 
studies should investigate whether writing samples provide reasonable 
substitutes for oral free production. 

Recommendations. Based on the pilot test results, we recommend that 
during the next stage of speaking test developmeni;: 

• the revised sentence repetition task be subject to validation trials; 

• a second sentence repetition task be developed and validated to 
generate a second form of the test; 

• a paragraph-paraphrase task be developed and piloted; 

• a story question-answer task be developed and piloted; 

• a second paragraph-paraphrase task or a second story question- 
answer task be produced and piloted depending on the outcome of the 
initial piloting of the two formats; 

• a training procedure be developed for raters that includes an overview 
of the scheme, examples of use, repeated cycles of practice and 
feedback, and reliability checks; 

• validation studies consider how best to reduce the time burden for 
administration and scoring. 



Summary 

This test development project has produced three prototype evaluation 
instruments, one assessing student writing and two assessing oral 
production. These instroments will help the LAUSD assess the effects of its 
new language arts curriculum for African American students on student 
proficiency in MAE. While this work is preliminary, the outcome of the June 
1991 pilot testing effort is promising. Both the revised Writing Assessment 
Scoring Scheme and the revised structured speaking test should be subje'^t to 
repeated validation trials. Two alternative formats also have been suggested to 



possibly replace the free production oral test. If the district believes that the 
administration time and the scoring time demanded by such formats are 
feasible, additional development and pilot testing of these alternatives will be 
required. Additional validation work is necessary before the instruments 
described here can be used for evaluation purposes. 



Appendix A 

Directions for the Test Administrator 
Writing Pilot Test, June 1991 




June 1991 

LAUSD Language Development Program 
for African American Students 

Class Cover Sheet 
WRITING 

Please provide the information requested in the spaces below to assist us in 
evaluating the writing samples from your smdents- 

Date 

Grade 

Teacher's name 

School 

When you have collected the writing samples from the students, please sort the 
papers into two groups: one, all African American student papers; two, all 
others. Please label the two groups* 

Please provide any comments you have about the writing procedure and/or 
topics on this sheet. Your feedback will help us refine the writing test. 




LAUSD Language Development Program 
for African American Students 



Directions to the Test Administrator 
WRITING 



The students should have two class periods to write their stories. Please allow 
30/40 minutes each day for the students to write. On the first day, the students 
should write as much of their stories as they can. On the second day, they 
should complete their stories and check them for errors. The directions to the 
students on the assignment sheets ask them to use Mainstream American 
English for their stories- 



Day One 

Pass out an assignment sheet and a writing sheet to each student in the class- 
Four topics are being equally distributed among the students- Because the 
topics are being pilot tested, some of the student will have picture topics and 
some will not. 

Say to the students : 

"I have just given you each a writing assignment. You do not all have 
the same assignment- Some of the assignments have pictures and some 
do not. Write the best story you can for your assigmnent using 
Mainstream American English not African American Language- 
Look at your assignment sheet and silently read your assignment. It is 
in bold type. ! will answer any questions you have about your 
assignment individually in just a minute." [Give the students time to 
read their assignments and then continue-] 

Sav to the students : 

"Now I want us to read the rest of the information on the sheet together. 




You will have two class periods to write your story, today and 
tomorrow. Today you should get as many ideas on paper as you can. 
Tomorrow you should complete your story and check it for errors. 

In your story, be sure to: 

- tell what happened and the order in which it happened, 

- give details about the situation, people, and events. Also tell how you 
felt about them, 

Before you begin your story, write your name at the top of this page 
and at the top of your story page. Write your name on the top of any 
additional pages you use. 

When you have finished writing your story, do not recopy it. Make any 
corrections you want to make right on the page just above the words or 
phrases you want to change. 

Do you have any questions?" 

If there are no questions, let the students begin. If there are general questions 
about the procedure, answer them. Circulate to answer questions about 
specific assignments. 

At the end of 30/40 minutes, collect all papers, assignment sheets and writing pag( 
Count to be sure you have all of them before you allow the students to leave. 

Day Two 

Pass out the assignment sheets and writing pages to the students. Remind them 
that they are to complete their stories and check their writing for errors. Also 
remind them that the stories are being written in Mainstream American 
English. 

When the time is over (30/40 minutes), collect all papers once again, both 
assignment sheets and writing pages. Count to be sure you have all of them 
before you allow the students to leave. 

Separate the papers into two groups-African American papers, all others- 
keeping the assignment sheet and writing page for each student together. 
Label the two groups. 

Give all testing material to the LDPAAS faciUtator/advisor. 
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Appendix B 



Topics for the Writing Pilot Test 
June 1991 



WRITING ASSIGNMENT 



Directions: Write about a very spec.il memory that made you 
happy. It could be something that happened to you^ something you 
did, or a place that you have gone. Use Mainstream American 
English to tell your story* 

You will have two class periods to write your story, today and tomorrow* 
Today you should get as many ideas on paper as you can. Tomorrow you 
should complete your story and check it for errors. 



In your story, be sure to: 

- tell what happened and the order in which it happened. 

- give details about the situation, people, and events. Also tell how you felt 
about them. 



Before you begin your story, write your name at the top of this page and at 
the top of your story page. Write your name on the top of any additional 
pages you use. 

When you have finished writing your story, do not recopy it Make any 
corrections you want to make right on the page just above the words or 
phrases you want to change. 



^ 3 ( ) 



WRITING ASSIGNMENT 



Directions: Wnle about a new friend you made this year. Tell 
how you met your friend and why you like your friend. Use 
Mainstream American English to tell your story. 

You will have two class periods to write your story, today and tomorrow. 
Today you should get as many ideas on paper as you can. Tomorrow you 
should complete your story and check it for errors. 

In your story, be sure to: 

- teU what happened and the order in which it happened. 

- five details about the situation, people, and events. Also tell how you felt 
about them. 



Before you begin your story, write your name at the top of this page and at 
the top of your story page. Write your name on the top of any additional 
pages you use. 

When you have finished writing your story, do not recopy it. Make any 
corrections you want to make right on the page just above the words or 
phrases you want to change. 
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WRITING ASSIGNMENT 




Directions: Look at the picture above* The boy's name is Bud* He has made a 
new friend, Using Mainstream American English, write a story about their first 
adventure. 

You will have two class periods to write your story, today and tomorrow. Today 
you should get as many ideas on paper as you can* Tomorrow you should complete 
your story and check it for errors. 



In your story, be sure to: 

- tell what happened and the order in which it happened- 

- give details about the situation, people, and events. Also tell how you felt about 
ihem. ' 



Before you begin your story, write your name at the top of this page and at the top 
of your story page. Write your name on the top of any additional pages you use. 

When you have finished writing your story, do not recopy it. Make any corrections 
you want to make right on the page just above the words or phrases you want to 
change. 
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WRITING ASSIGNMENT 



Directions: Look at the picture above. Who do you think the people m the 
picture are? Using Mainstream American English, write a story about what is 
happening. 

You will have two class periods to write your story, today and tomorrow. Today 
you should get as many ideas on paper as you can. Tomorrow you should complete 
your story and check it for errors. 

In your story, be sure to: 

- tell what happened and the order in which it happened. 

- give details about the situation, people, and events. Also tell how you felt about 
them. 

Before you begin your story, write your name at the top of this page and at the top 
of your story page. Write your name on the top of any additional pages you use. 

When you have finished writing your story, do not recopy it. Make any corrections 
you want to make right on the page just above the words or phrases you want to 
change. 

27 
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Appendix C 

Writing Samples for the Same Prompt 
Grades 3-6 



Feature occurrences are circled on the papers, and a scoring scheme is 
included for each paper. The scoring scheme is an interim version similar to 
the revised form. 
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34 



student E 



Grade 3 











a 

ERIC 



Writing Assessment Scoring Scheme 



Student's name ^K^-f t Date (//tlhi 

Topic Bey ftwid (a4- School t^l^^m Ui'K? 

Grade 3 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAP 








riuraiizmion 
plural mariner -s 


1 








) 


Possession 

possessive marker - s 




+ 








Past Tense 

past tense marker -ed 


mj tHi fKI in 


+ 




= 


1? 


Auxiliary/Present tense copula 

is/are - includinq contractions 










I 


Auxiliary/Past tense copula 
verbs 

wasAvere - including contractions 


11 








II 


Third person singular 
rrafter -s 












Indefinite article before vowels 
a/an 












Demonstrative pronoun 
them/those 




+ 








Reflexive pronoun 

himself/hisself themselves/lheirselt 




+ 








USAGE TOTAL 






0 







^ME Usage Total/Feature Total = ^^' I i-"^ = % 



Other Features: 



Evaluator's signature 
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Writing Assessment Scoring Scheme 



student's name ^dx^ F 



Date 



Topic g<y a\A^ M 



School B/?(^u?ivi Hills 
Grade *\ 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 




AAL 




Pluralization 
plural mariner -s 


1 






= 


1 


PossGSsion 

possessive marker -'s 










O 


Past Tense 

past tense marker -ed 


mi mam 


+ 








Auxiliary/Present tense copula 
verbs 

is/are - irKluding contractions 










0 


Auxiliary/Past tense copula 
verbs 

was/were including contractions 


ii 


+ 








Third person singular 
marter -s 




+ 






0 


Indefinite article before vowels 
a/an 




+ 






0 


Demonstrative pronoun 
thenVthose 




+ 






o 


Reflexive pronoun 

himself/hisself themselves/lheirsen 




+ 






0 


USAGE TOTAL 


18 


+ 


D 




18 



MAE Usage Total/Feature Total = _/| / fg = foo % 



Other Features: 



^.1 (} 



Evaluator's signature 



student G 



Grade 5 



^wr. 'JJ^Cgfe^O/ni 

(^'a ^cd kC^Jmr mo^ (km JA cd 
ifl cat ^^^02^ 





ERIC 
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Writing Assessment Scoring Sctit^me 



student's name. S^dm^ Ci Date i^/^ll^l 

Topic Ovid CaJt School ?>Ci\A\^'^ fhlU 

Grade ^ 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 1 


AAL 




Pliirflli7ation 

1 lUI wli&iClllwt 1 

plural marker -s 


\ 


+ 




= 


1 


r ossession 

possessive marlor -'s 


(Mi 






= 


M- 


rasi 1 Gnse 

past tense marf<er -ed 


m,l rni Hil 


+ 


nii 


= 




Auxiliary/Present tense copula 
verbs 

is/are - includina contractions 




+ 






Q 


Auxiliary/Past tense copula 
verbs 

was/were - including contractions 


rm u 


+ 






1 


Third person singular 
noarker -s 




+ 








Indefinite article before vowels 
a/an 


1 


+ 






\ 


Demonstrative pronoun 
Ifiem/those 




+ 






o 


Reflexive pronoun 

himselt/hisself themselves/lheirseH 




+ 






o 


USAGE TOTAL 


35 




f 




5^ 



MAE Usage Total/Feature Total = / 31 = ^t? % 



Other Features: 



Evaluator's signature C(^4 



student H 
Grade 6 









- " 37"* A A ^ 



O I 

ERIC 








JuJjj iuiyjJL ^fuyfL .u>4aa^m(^^ ^ 



38 



4:j 



Writing Assessment Scoring Scheme 

Student's name M Date (rfz-l j^l 

Topic Boy a\Ad School B^ldt^ti^ Hn«$ 



Grade U. 



GRAMMATICAL FEATURES 


USAG 


E 


FEATURE 
TOTAL 


MAE 




AAL 




Pluralization 
plural marker -s 


nn 










Possfission 

possessive marker 








— 


0 


Pa<%t Tfin^fl 

past tense marker -ed 


K44 \Hi 
TRi mi H41 


+ 




= 


So 


Auxiliary/Present tense copula 
verbs 

is/are - includinci contractions 


1 1 1 1 
u 1 \ 


+ 






4 


Auxiliaiy/Past tense copula 
verbs 

was/were - including contractions 


rm mi 
m 


+ 






(1 


Third person singular 
nnarker -s 




+ 






t? 


Indefinite article before vowels 
a/an 




+ 






0 


Demonstrative pronoun 
them/those 




+ 






0 


Reflexive pronoun 

himself/liisself themselvesAheirsell 




+ 






o 


USAGE TOTAL 




+ 


0 







MAE Usage Total/Feature Total = i S-B = (ov % 



Other Features: 



Evaluator's signature 




Appendix D 

Two Writing Samples Each from Grades 4 and 5: 
One sample based on a sentence prompt, one on a picture prompt 



Feature occurrences are circled on the papers, and a scoring scheme is 
included for each paper. The scoring scheme is an interim version similar to the 
revised form. 
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student A 
Grade 5 





^ /jm- ir M MUM(m>MJ^ 

(3ffc '/IWM- jMhkl^ pio, fjlw^ ft 





' ' ' WMy £mb mml 0m 

iwW/ i/v^ '^iU/ /YiJy^ 
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ERiC 



Writing Assessment Scoring Scheme 



Student's name A Date ojill^l 

Topic \^(^ ^o\Ac\ School Etfl^i^in 

Grade 5 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 




AAL 




Pluralization 
plural mariner -s 


W 








Z 


Possession 

possessive marker -'s 








= 


0 


Past Tense 

past tense marker -ed 


III) 






= 


4 


Auxiliary/Present tense copula 
verbs 

Is/are - including contractions 


Hii 7^W lili 










Auxiliary/Past tense copula 
verbs 

was/were - including contractions 


1 








/ 


Third person singular 

marker -s 


11 








z 


Indefinite article before vov^^els 
a/an 










0 


Demonstrative pronoun 
them/those 










o 


Reflexive pronoun 

himself/hisself themselves/theirsett 










0 


USAGE TOTAL 


11 




D 




23 



MAE Usage Total/Feature Total = / = loo % 



Other Features: 



Evaluate r's signature 




student B 
Grade 5 



i^i^Am owdi uWa JvV^ J^aojK^^ Jdf\t JQATvA jJnl (^>^iib^)JJlrtiUy^v^^rl/^^ 

- 43 ' 

5_Q 



ERIC 



0* q'.OO o-caoch. }^(^^^^rt\Drt^^^c^^^ 4J:f$^^^<^4^ 
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Writing Assessment Scoring Scheme 



student's name "y^diM^ €> Date 6^7-7^1 

Topic Cn'v\ ^\Ad ?£iveAA\^ School ^\d\x>\\A Hi'Hs 

Grade S 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 




AAL 


Pluralization 
plural maiicer -s 


mi 








G 


i woo w wOlWI 1 

possessive marker -'s 






1 




1 


past tense marker -ed 


fm rni mj mi 




1 


— 


51 


Auxiliary/Present tense copula 
verbs 

Is/are - including contractions 






1 






Auxiliary/Past tense copula 
verbs 

was/were - including contractions 


rnj itij 








IP 


Third person singular 
marker -s 


1 








1. 


Indefinite article before vowels 
a/an 










0 


Demonstrative pronoun 

thenn/those 










0 


Reflexive pronoun 

hlmseff/hlsself themselves/lheirseM 












USAGE TOTAL 


5g 


+ 


3 







MAE Usage Total/Feature Total = / 0\ = % 



Other Features: 



v3 2 



Evaluator's signature 



student C 



Grade 4 



One. c>io^ '^^32^ /r\€^r.cA 



Tin 



On "fine. So<^[hiC//.-C/'^>iy me^ oi P>a 



jO^^ c^.noA -ry>y bor"f. la^r 



y W^O 

CvTvnc^ roo/v^ So\/\/e c. '.\|nr nswetr- 




46 
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Writing Assessment Scoring Scheme 



student's name Sh^-hC Date o/ ^7/ i 

Topic S^cia\ McwioKy School f]*j[M^ei ^jvuA- 

Grade ^ 



GRAMMATICAL FEATURES 


USAGE 


FFATIJRF 
TOTAL 


MAC 1 1 AAL 




Pluralization 
pkirai marker -s 


111 


+ 


1 






Possession 

possessive marker -*s 




+ 


1 




I 


Past Tense 

past tense marker -ed 




+ 


I 




s 


Auxiliary/Present tense copula 
verbs 

is/are - irK:luding contractions 


11 


+ 






9 


Auxiliary/Past tense copula 
verbs 

was/were - inciuding contractions 


1 


+ 






I 


Third person singular 
marker -s 










0 


Indefinite article before vowels 
a/an 




+ 






0 


Demonstrative pronoun 
them/those 










0 


Reflexive pronoun 

himsetf/hisseff themselves/theirseH 




+ 






0 


USAGE TOTAL 


10 


+ 


3 




13 



MAE Usage Total/Feature Total = / = 77 % 



Other Features: 



Evaluator*s signature 

^ 47 



student D 



Oymjl- wiA-fWhTw^ ockvYYU, <^ hcrt^ { {T^^^ 



OruL (jX^^ ^.<i-imxc ^'^^^ xya<itm. 

.,H^(>tiX >/a(^^ :!tkc^ J<jl AaJ^^ ^ft^ ^drt^ .yiirT- 



; ^jtl, ..Mt^ Jl>txci/t<^ JIka^ .^yvtvo.^ ..^^ . 

JLhM^ AVfir- .xyU^-Wv- ^-rt-Cc^ X-^^C^tUoAJL JM^ ^-j^f^t^^ ^ g*-^ <^ 

x<icr ^x)VrJ^> 3^t>>t^ ^W)-*^ OaH^ J^Jti 

Jt^oJe Cc/v^A^ (^o^^^) ^W-uu -^-ooc jt^r-^^^ JXtiJl 

oj^ zfj^ cLO\^ cxA-sftL ^^^^ (y\<y^ (yrJL 



55 



48 



ERIC 



Writing Assessment Scoring Sclieme 



Student's name Sht^-h D Date U/z-llH 

Topic Boy (CtA- School Pi^l^vod ?-hr<^4- 

Grade ^ 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 1 


A A t 

AAL 




Pluralization 
plural marker -s 






1 




i 


Possession 

possessive marker -'s 










0 


Past Tense 

past tense marker -ed 


rm mi nw II 


+ 


H44 111 






Auxiliary/Present tense copula 
verbs 

is/are - Includinq contractions 


HII 


+ 






4 


Auxiliary/Past tense copula 
verbs 

was.AA/ere - including contractions 


11 


+ 






2. 


Third person singular 
marker -s 




+ 


I 




I 


Indefinite article before vowels 
a/an 




+ 






D 


Demonstrative pronoun 
them/those 




+ 






0 


Reflexive pronoun 

himself/hissetf ihemselves/theirsetf 










o 


USAGE TOTAL 






10 




33 



MAE Usage Total/Feature Total = / 3s ^ Id % 



Other Features: 



Evaluator's signature 




Appendix E 

Writing Assessment Scoring Scheme: 
Initial draft and revised form 



D ( 

50 



Tentative scoring scheme: Research model 
Student's name 



Date 



WRITING 



^ D A tf IJ A *T*I/^ A 1 CCA *T*I ID CO 

GnAMMATICAL rcATUHcS 


USAGE 


MAE 


AAL 


POSSIBLE OCCURRENCES 


Pluralization 








Possession 








Past tense 








Auxiliary verbs 








Third person singular 








Past tense copula verbs 








Present tense copula verbs 








Second person plural 








Indefinite article 








Demonstrative pronoun 








Reflexive pronoun 









Comments: 



Eivaluator's signature 



ERiC 
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Writing Assessment Scoring Scheme for Use of MAE 



Student's name 
Topic . 



Date 
School _ 
Grade 



GRAMMATICAL FEATURES 


USAGE 


FEATURE 
TOTAL 


MAE 


OTHER 1 


Pluralization 
plural marker -s 


$ 










r U\>oc>oolUi 1 

possessive marker -'s 


c 










Past tense 

past tense marker -ed 


ed 










Present tense copula auxiliary 
verbs 

am/is/are - including contractions 


is/are/anrV's/Ye/'m 


+ 








Past tense copula/auxiliary 
verbs 

was/were - including contractions 


were 


+ 


was 






Third person singular 
marker -s 


s 


+ 


a 






Indefinite article before vowels 
an/a 


an 




a 






Demonstrative pronoun 
those/them 


those 




them 






Reflexive pronoun 

hlmself/hisself themselves/theirself 


himself/themselves 




hisself/theirself 






USAGE GRAND TOTAL 




+ 









MAE Usage Grand Total/Feature Grand Total = 



% 



Other Features: 



Evaluator's signature 
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Appendix F 

Errors From Pretest Writing Samples 
Not Readily Attributable to Differences 
Between MAE and ALL 



0 
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Form in 
Students* 

Essays 

a 

me 

us 

us 

weVe 
it 



Corrected 
Fgrm 

I 
I 

me 
we 
we 
them 



my 

her's 

are 

movies 

aunts 

someones 

friends 

dads 

parents 

Ambers 

Roberts 



mme 

hers 

our 

movie's 

aunt's 

someone's 

friend's 

dad's 

parents' 

Amber's 

Robert's 



were 
that 
an 
my 



alway 
sometime 
it seem 
lot 

happen 



where 
at 
and 
why 



always 
sometimes 
it seems 
lots 

happens 
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Form in 
Students* 

did not have nothing* 



Corrected 
Form 

did not have anything 



no more* 


anymore 


none* 


any 


know more 


anymore 


goto 


to go to 


was gone 


was going 


look 


looking/watching 


gaves 


gives 


at's 


acts 


is* 


are 


have got 


has 


got 


have 


know 


known 


went 


rode 


hanged 


hung 


was to go 


went 


we're 


were 


fell 


felt 


wakes 


woke 


yells 


yelled 


starts 


started 


nags 


nagged 


ignores 


ignored 


can 


came 


had feeled 


had felt 


had got 


had gotten 


had rided 


had ridden 


was very contented 


was very content 


was broke 


was broken 


got morning time 


was/became 


don't can go 


can't go 


don't got 


don't 


went 


went to 


had went 


had gone 


got bit 


got bitten 


could 


could have 


this 


there was 
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Form in 
Students' 

Essays 

it 

their 
passit 



Corrected 
Form 

there were 
they were 
past it 



111 

lets 

where 

were 

thats 

He's is 

wont 

your 

whats 



rii 

let's 

we're 

we're 

that's 

He's/He is 

won't 

you're 

what's 



* AAL forms which vary from MAE but are not under investigation in this study 
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Appendix G 

Directions for the Test Administrator and 
Scoring Sheet for the Structured Task 

Speaking Pilot Test, June 1991 
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B4 



L AUSD Language Development Program 
for African American Students 

Directions to the Test Administrator 

SPEAKING 



There are two parts to the Speaking test. Part I is a structured task. Part II is 
a free production task. The same students should do both parts of the test. 
Please select four students from your class who you feel represent a range of 
language ability. 

Parti: Structured Task 

You will read ten sentences to the student one at a time and the student will 
repeat each sentence. You should read each sentence to the student only once. 
This procedure will be tape recorded. 

For each sentence, one or more points of difference between Mainstream 
American English (MAE) and African American Language (AAL) are being 
tested. You are to listen for the student's use of MAE. If the student uses 
MAE for the point tested, do not write anything on the scoring sheet. If the 
student uses AAL, check that s/he has done so on the line which corresponds 
to the point being tested. (See attached scoring sheet.) 



Say to the student : 

"I am going to read ten sentences to you one at a time. I want 
you to repeat each sentence exactly as I say it. I will say 
each sentence only one time. I will record what you say so we 
can hear it again later. Do you have any questions?" 

[Go to the scoring sheet for the sentences.] 

Before you begin, be sure to write the student's name, grade in school, and 
the date in the spaces provided on the scoring sheet. 

When you and the student are ready, start the cassette recorder. 
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Sav to the student : 



"Tell me your name/* 

Then begin the procedure. When the student has repeated the last sentence, 
stop the recorder and sign the bottom of the scoring sheet. 

Please provide any comments you have about the structured task on this 
sheet. You comments will help us refine the speaking test. 
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Scoring Sheet: SPEAKING 



Student's name Date 



^ Part I: Structured Task 

Directions: If the student uses MAE, do not write anything. If the student uses AAL, 
check on the appropriate line. 







• plural. 




The two pencil^ are on the desk. 


• sound cluster 
(word final) 






• Dossessive 


• 


John's car is old 


• the /r/ sound 

• sound cluster 

(word final) 


# 


The police are in the store. 


• stress 




She!£ a very smart girl. 


• auxiliary verb (contraction) 




The little boy hurt himself when he fell. 


• reflexive pronoun 




We were aoing on a picnic, but it started 
to rain. 


• past tense copula verb 


• 


Jessie has an aunt who lives next door. 


• indefinite article 






• the /th/ sound 




Your brother called me yesterday. 


• sound cluster /Id/ 




Bud plavs football after school every day. 


• third person singular 




1 sm very happy because 1 have a kitten. 


• present tense copula verb 


# 


Evaluator's signature 
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Appendix H 

Revised Scoring Sheet 
for the Structured Task 
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Scoring Sheet: SPEAKING 



Student's name Date 



Grade 

Structured Task Pretest 



Ask the student to repeat the words below. If the student produces the underlined sound in 
each word, put a check on the line. If the soident does not produce the sound, put a zero (6) 
on the line. 

clothes M 

niouse /s/ 

choices M 



house 


/s/ 


glasses 




shoes 


M 


dress 


/s/ 


brushes 


M 



toys I2J 



Show the student the two pictures one at a time and say: 
Picture #1 Show me and tell me which one is taller, 
(picture of a man and a boy) 

The student should point to the tall man and say "He is." If the student does not say "He is," 
write down what he/she says. If the student answers correctly, do not write anything. 

Picture #2 Stiow me and tell me which one is older, 
(picture of a grandmother and a younger woman) 

The student should point to the grandmother and say "She is." If the student does not say "She 
is," write down what he/she says. If the student answers correctly, do not write anything. 
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structured Task 



Directions: Ask the student to repeat each sentence. If the student uses MAE, put a check on 
the appropriate line, K the student does not use MAE, put a zero (0) on the line. 



The two cats are chasing the rat 


• plural 


W hen my sister has extra money, 
sne Duys me loe crearrL 


• third person singular 


I Saw mi appil; Wlul a WOriii 111 li» 


lllUvllllllC iU Llvlv 


Mv cousin hurt himself when he fell 
Oil uie noTse* 


• reflexive pronoun 


We were going to play video games, 
Dui we Ulan i nave enougfi money* 


• past tense auxiliary verb 


I love my grandmother. Sheis wonderful. 


• copula verb fcontraction"! 


My brotherls puppy ate my sock. 


• possessive 


We atfi excited because there will 
be a party at school tomorrow. 


• present tense copula verb 


The little boy plays every day with his 
two kitty cat friendj. 


• third person singular 

• plural 


You^ll have a pretty smile if you take 
care of your teelh. 


• the /thy sound 


Hat mother bird laid an egg in my flower pot 


• the /th/ sound 

• indefinite article 


Bud^ a sharp dude. 


• copula verb f con traction) 


The Lakers are alwavs proud of themselves 
when they win a basketball game. 


• reflexive pronoun 


I an) very happy because I have good friends. 


• present tense copula verb 


Ihey were going to the beach, but the car 
had a flat tire. 


• the /th/ sound 

• past tense auxiliary verb 


Billyi cousin always says, "Thank you," when 
someone is polite to her. 


• possessive 

• the /th/ sound 


Evaluator's signature 


Scc«B 
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Appendix I 

Directions for the Test Administrator and 
Pictures and Oral Prompts for the Free Production Task 

Speaking Pilot Test, June 1991 
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PartH: Free Production Task 



For this task give each of the four students a different picture (A, B, C, or D) 
to serve as a basis for telling a story to a small group of classmates. 



give thg picture to the student and ^9y: 

"Here is a picture. I want you to tell a story about the picture 
to a small group of your classmates. Use Mainstream American 
English to tell your story. I will record the story so we can hear it 
again later." 

Read aloud to the student the two-sentence oral prompt that corresponds to 
the picture- A, B, C, or D. 

Oral Prompts for the Free Production Task 

A. This is Willie. He found a secret door to a secret place in his 
neighborhood. Tell about what you think he found. 

B. These children are ready for an adventure. Where do you 
think they are going and what are they going to do? 

C. Who do you think the person in the picture is? Tell about what 
is happening. 

D. Who do you tliink the person in the picture is? Tell about what 
is happening. 

Say to the student : 

"Take some time to think about your story while I get your 
classmates together," 

While the student is planning his/her story, bring the small group of 
classmates (3-5 students) together. (Allow approximately 1 minute.) 

Wfitn the group is settled and the student is ready to begin, start the cassette 
recorder and let the student tell his/her story. When the student has finished, 
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state the student's name, grade level, and the date^ so the information will be 
on the tape. Then turn off the recorder. 



Give all testing material to the facilitator/advisor. 

Please provide any comments you have about the free production task and 
topics on this sheet. Your comments wiU help us refine the speaking test 
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Appendix J 



Draft Scoring Sheet for the Free Production Task 



00 



73 



ERiC 



Tentative scoring scheme: Research model 

Student's name Date 



SPEAKING (Part il) 







USAGE 






MAE 


AAL 1 


POSSIBLE OCCURRENCES 


• 


Sound clusters 
Beginning of word 










MiQQie OT woro 








• 


Word final 










Stress patterns 










First person future 








• 


Multiple negation 










Use of GO 










GRAMMATICAL FEATURES 


MAP 


AAI 


po^siRi P nnni jRRPNriPf? 




riuraiizaiion 










r OooeSolOiI 








• 


rBsi lense 










Muxfiiary veros 










inira person singular 








• 


Past tense coduIa vprbs 










Present tense copula verbs 








• 


Second person plural 








Indefinite article 










Demonstrative pronoun 








• 


Reflexive pronoun 
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ERiC 



Tentative scoring scheme: Research model 
Student's name 

SPEAKING (Part II) 

Comments: 



Evaluator's signature 



Appendix K 

Transcribed Responses per Grade Level 
for Each of the Four Prompts in the Free Production Task 

Speaking Pilot Test, June 1991 
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Prompt jQ: Who do you think the person in the picture is? Tell about what is 
happening. 

(K) Someone is riding the horse and the peaches are gonna fall on 'em and he's 
running away. 

(2) The title of my story is "Ivy Yoyo." Ivy Yoyo was on a horse and he was 

running from two pumpkin and then the horse was taking him somewhere 
and when they went somewhere they had fun. 

(4) Once upon a time there lived a man. He lived in a place called Pumpkin 
Town. He lived in a pumpkin house. He had a horse and the people called 
him Peter Pumpkin Eater. Everybody liked him. The end. 

(4) In the picture I see a little boy on the horse riding away from a big giant 

pumpkin that's chasin' him. He's trying to get away as fast as he can, but he 
can come home safe. The horse doesn't have no idea what's happening but 
he's getting a message to hurry as fast as he can. He makes it home very 
safe and tells his whole family what happened the same day. The end. He 
lived happily ever after. 

(6) I think the person in the picture is a little boy and he looks like he's nmning 
away from giant pumpkins that grew when he was growing um, and he's 
running through a farm, a farm yard with his horse and he looks very 
scared. 

(6) Once upon a time there was a boy and his horse. Billy was the boy's name. 
Thunder was the boy 3 horse. Billy and Thunder rode through a pimapkin 
patch, but Thunder's hooves got stuck in the leaves so all of a sudden 
Thunder sees big clouds in the sky and so does Billy. The clouds go overhead 
then all of a sudden the sun comes back out, so Billy and Thunder ride back 
home. 



Prompt n* Who do you think the person in the picture is? Tell about what is 
happening. 

(K) The little girl walkin' walk with flowers in her hand. She saw a 
(incomprehensible). 

(2) I think the person in the picture is Simmier and and what is happening is a 
a girl is picking flowers in the forest and a and a wolf is spying on her in the 
forest. 

(4) Once upon a time there was, there lived a girl in a small, in a small house by 
the forest. She would always visit her grandmother on the other side of the 
forest. But one day she wanted to give her grandmother some flowers. She 
was walking and she lost the trail and then a himgry wolf came upon her 



84 

77 



The grade is provided in the parenthesis next to the transcribed speech of the 
student. 



Prompt A* This is Willie. He found a secret door to a secret place in his 
neighborhood. Tell about what you think he found. 

(K) He saw a door and he push his way out. 

(2) I think Willie found a kitten or a ball. All the people were looking at him. 

(2) I think he found the secret place imder the gate into a special place where he 
was. 

(4) One day Willie found a secret door in his neighborhood and then when he got 
there he found a frog, and the back of a gate and he told all of his friends 
when he got to school the next day. 

(6) I think Willie found a secret hideout place in his backyard under a wall in 
the grass where he goes down in after school and at night time. 



Prompt These children are ready for an adventure. Where do you think they 
are going and what are they going to do? 

(K) They goin* to Magic Mountain and havin' a lot of fun playin' on the rides and 
getting into the water. 

(2) These kids are going to Mars ^d um they probably see some creatures that 
try to come after em and try to get em and eat em up. And they would take 
em away and lun they never see their mother or father again and they try to 
uh get away from em but they can*t cause they got long arms and legs like uh 
pull they tongue out and they catch em and eat em up with salt, sugar and all 
that stuff on em and cook em up and they 11 v^at em. 

(4) Once upon a time there were Cwe kids who lived down the street from each 
other named Tony, John, Kim, Kimberly, and Lynn. They, they were 
planning a picnic for late Friday to plan their Mother*s Day gift, and Tom, he 
made up the whole idea to give tl>eir parent a big girafte and share it. The 
end. 

(6) I think these children are going from door to door because it s Halloween and 
and they're dressed up funny and they want them, and they want people to 
notice them that that it is Halloween and they're... 



and growled at her. She started to nin> but she wasn't goin' the right way 
but until she uh fotmd, but until she found a trail, 8he» she followed it all the 
way down to her grandmother s where the hungry wolf was waiting. She, 
she came in. The himgry wolf jumped at her. She, she [caud] called for help 
even though it was useful but a hunter that knew her grandmother came 
over and shot the wolf and that was the end of the hungry wolf. The end. 

Once upon a time there was a little girl. She was walkin* in the woods 
pickin* flowers for her mother. She xim she was pickin' flowers for her 
mother. Suddenly she heard a noise. It was a a coyote and the coyote came 
up to her and she was so frightened she didn't know what to do. So she just 
stood there. Then after awhile she started thinking that if she, if she ignores 
him then he, he will not mess with her. So that is what she did and the 
coyote just went away. The end. 
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Introduction 



This document reports on the 2nd year of a 3-year project to develop 
evaluation instruments for assessing student proficiency in Mainstream 
American English (MAE) in speaking (Grades K-6) and writing (Grades 3-6). 
The instruments are being developed by the UCLA Center for the Study of 
Evaluation (CSE) for use in the Los Angeles Unified School District (LAUSD) 
Language Development Program for African American Students (LDPAAS). 
The LDPAAS provides teachers with instructional strategies for teaching 
MAE to speakers of African American Language (AAL) (Los Angeles Unified 
School District, forthcoming). The evaluation instruments being developed 
will be used with the program as pre and post measures to help determine 
program effectiveness at teaching MAE to speakers of AAL. For this reason, 
two forms of each instrument are being produced. 

During the. 1st year (1990-91), prototype instruments were developed for 
both speaking and writing (Butler, Herman, & Yamaguchi, 1991). During the 
2nd year, activities focused on the revision and refinement of the speaking 
instrument and the development of a second form of the instrument. The two 
forms of the speaking instrument. Form A and Form B, were pilot tested to 
determine equivalence and, in addition, potential new writing prompts were 
generated. The writing instrument and companion scoring procedure require 
revision and refinement, and two forms must be produced. Then both the 
speaking and writing instruments must be validated. These tasks will be 
conducted during Year 3. Until the validation cycle has been completed, the 
instruments cannot be used for evaluation purposes. 



This paper is organized in the following way. First, modifications in the 
prototype speaking test are discussed including development of a new format. 
Then the pilot testing procedure for establishing equivalence of forms is 
presented. Next there is a discussion of the new writing prompts. Finally, 
there is a summary section. 

Before beginning the specific discussions listed above, an important 
point from the Ist-year report (Butler et al., 1991) is repeated to provide context. 
With both the speaking and writing instruments, the content focus is on the 
differences between AAL and MAE. It is important to remember that the 
items on the tests address only specific linguistic differences between the two 
language systems, and therefore no broader interpretation of the students' 
language abilities should be inferred. The content base for the instruments is 
the section of the instructional strategies document (Los Angeles Unified 
School District, forthcoming) entitled: "Characteristic Linguistic Features of 
African American Language Contrasted with Linguistic Features of 
Mainstream American English." 



Speaking Test: Modifications in the Prototype 

During initial prototype development in Year 1, two formats for 
speaking were piloted: a structured test and a free production test. The 
structured test, in which the test administrator read sentences to each student 
and the student repeated what the test administrator said, allowed us to use 
exactly the same measure with each student. In that way, we were able to 
control the language environment and thus guarantee opportunities for 
producing specific language features. Even though students were asked to 
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imitate sentences in MAE, we did elicit AAL from those students who were 
least proficient in MAE. The structured format and procedure was extremely 
successful. We concluded that the structured test prototype should be retained 
as part of the final speaking test. 

The free production test, on the other hand, proved to be problematic. In 
this task, each student was given a picture and a verbal prompt and asked to 
tell a story using MAE based on the picture. With the exception of four 
students in one kindergarten class, all of the students tested (K, 2, 4, 6) 
understood the task; however, the language samples produced were too brief to 
allow us to draw reliable conclusions about the students' use of MAE. Because 
the structured task did not prove to be an effective means of obtaining a ratable 
sample of speech with the population tested, we decided to consider alternative 
formats fcr the speaking test. 

As part of the prototype revision process, Lorraine Cole, Director of the 
Office of Minority Concerns for the American Speech-Language-Hearing 
Association, met as a consultant with Noma LeMoine, Director of the LAUSD 
LDPAAS, and the UCLA test development staff in July 1992 to discuss test 
content issues. 

As a result of the meeting, changes were undertaken to refine the 
structured test, and a new format was developed to replace the free production 
test. Also, as part of our discussions, 10 feature differences between MAE and 
AAL were selected from the features on the initial structured test and others 
suggested by Cole for inclusion on the revised prototype to be piloted in August 
1992. Those features were: 



1. plural 

2. demonstrative 

3. third person singular 

4. reflexive 

5. past "was/were" 

6. present "is/are" 

7. possessive 

8. wh-locative 

9. negation 

10. distributive *'be" 



See Appendix A for an example of each feature. These 10 features were 
identified on the basij of three criteria: (a) they are among the most frequently 
noted differences between the two language systems, (b) they are readily 
identifiable as features of AAL by speakers of MAE, and (c) they are features 
that can be easily and accurately assessed by ev^luators who are linguistically 
unsophisticated. See Butler et al. (1991, pp. 12-14) for an earlier discussion of 
feature selection. 

Structured Speaking Test 

While the structured speaking test performed well during Year 1 
piloting, the individual sentences were revised after the piloting at Cole's 
suggestion to make the content more interesting for the target population 
(Butler et al., 1991, p. 14). However, at the July meeting, the participants 
agreed that the prototype could be further improved in the direction of making 
the content more engaging for the students by having the individually repeated 
sentences comprise a story .ather than be a collection of unrelated sentences. 

To that end, two story lines were developed using sentences containing 
the 10 feature differences between MAE and AAL discussed above. Because 
the format was essentially unchanged in that the sentences were simply 
linked by a story, the revised structured task was not included in the August 
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piloting. Rather, one story was included on Form A of the test and one on 
Form B for the equivalence testing in October 1992. For each form, there are 20 
sentences in the structured section with each sentence testing one feature. 
Each of the 10 features is tested twice in the structured section. 

New Speaking Test Format 

Because the free production prototype piloted during Year 1 failed to 
produce ratable language samples from the students, we decided to develop a 
format in which the language environment would be more controlled and that 
would, we hoped, elicit the 10 feature differences between MAE and AAL listed 
above. At the same time, we wanted to be sure the task was not purely 
imitative because we already had a successful format of that type in place. 

The format developed for pilot testing consisted of a series of pictures 
illustrating a story with companion questions that were based on the story 
content and that focused on the 10 feature differences between MAE and AAL 
identified as most salient for our purposes. The administration procedure 
involved having the test administrator, in this case, the classroom teacher, 
point to a picture, read a specified portion of the story, and then ask the student 
questions about the portion of the story just read. By presenting the story 
segment by segment, we hoped to eliminate any short-term memory effects. 

Two types of picture prompts were used with this story format. One type 
consisted of a four-frame cartoon with all of the frames on one page. One 
cartoon was a modified "Calvin and Hobbes" cartoon, and the other was a 
modified "Garfield" cartoon. The second type of picture prompt involved a 
series of individual pictures, each on a separate page. The pictures were 
drawn by an artist to match two story lines developed by CSE staff, One story 



line was about a little girl named LaToya and her younger brother Tevin. The 
second was about Sturbridge the Pigeon. 

Au gust 1992 Pilot Testing 

The purpose of the next phase of pilot testing, which was conducted in 
August 1992, was to ascertain whether the new format for the speaking test 
would provide ratable language samples that would supplement the 
information from the structured task. In addition, the pilot testing would 
allow us to determine which type of picture prompt — cartoon or series of 
individual pictures — was more effective with the population being tested. 

We were also concerned about whether we could successfully elicit 
specific features. Three features in particular were problematic from an item 
writing perspective — demonstrative, reflexive, and distributive "be." It was 
extremely difficult with these three features to construct items for which the 
desired structure would be the most likely response. 

Table 1 shows the distribution of the features across the four story lines 
on the August pilot test: cartoon — Garfield, Calvin; picture series — Tevin and 
LaToya, Sturbridge, for a total of 38 items on the test. While some features 
proved easier to tap than others, distributive "be" was the only feature with just 
a single opportunity for production on the pilot test. 

The August 1992 pilot testing effort took place in Grades K, 2, 4, and 6. 
Most of the testing was conducted at three schools in the district, 52nd Street 
School, Manhattan Place School, and Figueroa Street School. However, 
because Manhattan Place School and Figueroa Street School do not have 6th 
grade classes, testing at the 6th grade level also took place at Normandie 
Avenue School. Table 2 shows the number of students tested per class per 
school. 
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Table 1 

Feature Distribution Across Cartoons and Picture Series for Speaking Test Prototype 
August 1992 Pilot 



Tevin and 

Garfield Calvin LaToya Sturbridge Row Total 



Plural 


1 


1 


2 


2 


6 


Demonstrative 


0 


1 




1 


3 


Third Person 
Singular 


1 


2 




1 


5 


Reflexive 


1 


0 




1 


3 


Past "was/were" 


1 


0 




2 


4 


Present "is/are" 


1 


1 




0 


3 


Possessive 


2 


1 


2 


2 


7 


Wh-locative 


2 


1 


0 


0 


3 


Negation 


2 


0 


0 


1 


3 


Distributive "be" 


0 


0 


1 


0 


1 


Column Total 


11 


7 


10 


10 


38 



Table 2 

Number of Students Tested bv Grade and School for Sneaking Test 
Prototype. August 1992 Pilot 

Grade and Number 

K 2,4 6 

4 4 4 4 

4 4 4 — 

4 4 4 — 

_ _ _ ga 



School 1 

(52nd St. School) 
School 2 

(Manhattan Place) 

School 3 
(Figueroa St.) 

School 4 

(Normandie Ave.) 



Note, — = no students tested. 
^ Two classes, four students each. 



The students at each grade came from the same class except for 
Normandie Avenue School Grade 6. A small number of students per class 
was tested so that the testing would not be a burden Lo the teachers. 

The teachers were asked to select for testing the four least proficient 
speakers of MAE in their classes, preferably two girls and two boys. We felt 
that in this way we would be better able to determine if the test items would 
elicit AAL because the students who are less proficient in MAE would be more 
likely to respond in AAL than would the more proficient MAE speakers. The 
test directions asked the student to answer the questions using MAE. 

Classroom teachers were briefed by a LDPAAS advisor on how to 
administer the test. CSE test development staff familiarized the advisor with 
the test administration procedures and stressed the importance of 
encouraging the teachers to provide feedback about the process and test content 
by completing a short questionnaire; Appendix B provides the "Directions for 
Test Administrators." Appendix C provides a copy of the feedback 
questionnaire. Of the 12 teachers who administered the test, 9 completed the 
questionnaire. Their comments are discussed below. 

The teachers administered the test individually to each student and 
scored the test on the spot. All of the tests were tape recorded to facilitate 
analysis of test results. Each response was judged only for the feature being 
tested and was assessed as being MAE, AAL, or Feature Not Present (FNP). 
FNP was selected when the response did not include the feature being tested. 

For example, one item with the Garfield cartoon involved a frame with 
Garfield and Jon sitting in a tree. The item tested for the feature "reflexive" by 
asking, "Who do you think put them there?" A MAE response was: "They put 
themselves there." An AAL response was: "They put theyself there." Two 
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FNP responses were: (1) "Garfield did" and (2) "They climbed up the tree." In 
both FNP cases, the feature being tested was not produced. 

To assist the teachers in making judgments about student responses, 
example responses were provided for MAE and AAL for each item on the test. 
For example, in the Garfield cartoon, the first frame showed Garfield sitting 
on his owner's (Jon's) lap with both of them watching T.V. In the test booklet 
the test administrator saw: 

Cartoon: Garfield 

Point to the first picture and say: 'This is Jon and his cat Garfield. They're 
watcliing their favorite music videos on T.V. What did I say they^re doing?*' 



RESPONSES: 


PRESENT "TS/AR3-1 






MAE 


AAL 






They are... 


Thev watchinET... 






TheyVe... 












FNP 


1. PRESENT 
"IS/ARE" 









Figure 1. Example responses provided in test booklet for the feature PRESENT "IS/ARE." 



The test administrator was to put a check in the appropriate box under the 
relevant example(s). For the example above, if a student responded, "They 
watching T.V.,** a check should have been put in the box under AAL. If a 
student responded with only "Watching FNP should have been checked 

because while the answer is appropriate in terms of its meaning, it does not 
provide enough language to determine whether the student would produce the 
feature being tested (Present "is/are") in MAE or AAL. 
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We knew that by combining the two cartoon stories and the two stories 
based on a series of pictures in one test, this pilot test would take longer to 
administer than the final speaking test. In spite of the test length, the 
students in all four grades maintained a high level of interest throughout the 
test. 

Pilot Testing Results 

Overall, the August pilot testing effort went very well. We were able to 
obtain the information we needed to make informed decisions about the final 
makeup of the speaking test. 

CSE test development staff listened to the recordings of all 48 speaking 
tests and used a special scoring form (see Appendix D) to make specific 
comments about student responses and to indicate whether the teachers had 
assessed the answers accurately. A critical question was whether the 
students would produce a desired response for a given prompt, that is, the 
feature being tested. In other words, would we obtain samples of speech 
ratable in terms of the features we wanted to assess? 

Another important issue had to do with identifying the type of picture 
prompt — cartoon or series of individual pictures — that would work best with 
the population being tested. A third critical issue was whether the teachers 
would, with the limited training they received, be able to make accurate 
judgments regarding the students' language use. 

Effectiveness of test iteir s. In general, the test items elicited desired 
responses from the students, that is, a given feature in either MAE or AAL. 
As expected, the items testing for a demonstrative form, a reflexive form, and 
distributive aspect yielded a substantially higher number of FNP ratings than 
did items testing for most other features. As mentioned above, it was 
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extremely difficult with those three features to generate items that would force 
a response only using the desired feature. On the basis of these results, we 
decided to eliminate the three features from the picture/story section of the 
final form of the speaking test and test them only through the repetition task. 
In spite of this decision, we ended up including one item that tested 
distributive "be" on Form B because it looked extremely promising. 

The one additional problematic feature on the August 1992 pilot was 
third person singular. There were five items in the test that attempted to elicit 
that form. Two of the five items received an unexpectedly high number of FNP 
ratings. Upon further scrutiny of all five third person singular items, we 
determined that the two problematic items lent themselves to more logical 
answers using other grammatical forms and were thus poorly constructed to 
test for third person singular. Since the remaining three third person 
singular items worked well, we decided to retain the feature on the 
picture/story section of the final speaking test. 

Type of picture prompt for final test form . Both the cartoons and the 
series of pictures used on the August 1992 pilot test elicited ratable responses 
from the students. Teacher comments on the questionnaire indicated that the 
students were able to follow and understand both formats. A few individual 
teachers noted a preference for one format over the other, but there was no 
discernible pattern. Since both formats appeared to work equally well, we 
decided to use the picture series as the new format for the speaking test. 
Having a series of pictures would facilitate a modification in the test 
administration procedure that would make administering the test less 
cumbersome. 

Revised test administration procedure . For the August 1992 pilot 
testing, the test administrator used a separate test booklet for each student. 
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The booklet included (a) the questions for the cartoon, (b) the script and 
questions for the story, and (c) example MAE and AAL answers for all 
questions. The student's answers were recorded in the booklet, page by page. 
The cartoons and the pictures that accompanied the stories were on separate 
sheets the test administrator had to manipulate along with the test booklet. 

In an attempt to make the test administration procedure more 
manageable, we reduced the size of the test booklet by producing laminated 
sheets with a picture on one side and the corresponding portion of the story 
and questions on the other side, thus eliminating the picture/story section 
from the test booklet with the exception of a single-page response sheet. (See 
Appendix E for the response sheets for the Form A and Form B picture/story 
sections.) 

The revised speaking test booklet now includes only the pretest, the 
structured section of the test, and the single-page response sheet for the 
picture/story section. The more compact test booklet with the laminated 
picture/story section has streamlined the speaking test procedure. 

Teacher/tester judgments . It became clear in reviewing the recorded 
tests that the level of training provided for the teachers was insufficient for 
assuring reliability and validity of test results. There was considerable 
variation in how the teachers administered the test even though they all had 
the same written test administration directions. Some teachers repeated 
questions several times if a student did not answer or did not provide the 
desired response. Some teachers reread parts of the story several times and 
would say, ''Listen while I read the story again." Still others would turn off the 
tape recorder, say something to the student, and then turn the recorder on 
again. Invariably the student would provide the desired MAE response. 
Clearly, a uniform test administration procedure was not followed for the 
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August pilot even though test administration directions were provided and a 
brief training conducted. One of the teachers commented on the questionnaire 
that "teachers need more explanation before giving the test." Indeed, much of 
the lack of uniformity on the August pilot was probably due to the limited 
amount of time available for training the teachers who were already stretched 
to the limit with their regular daily schedules. This issue of teacher/tester 
training was addressed during the October 1992 pilot. A 1-day training 
workshop was designed and conducted for that pilot by UCLA test development 
staff. 

Remaining issues for content modification . Suggestions from teachers 
regarding specific pictures and items were taken into consideration, and 
changes were made based on the suggestions and on test results. A change 
was made, for example, in the name of the pigeon who is the main character 
in one of the stories. In the August pilot, the pigeon's name was Sturbridge. 
Two teachers indicated that the name was too difficult for the younger children 
(K) to remember, so the pigeon's name was changed to Gavin for the October 
pilot. 

Discussion 

The results of the August 1992 pilot testing effort led to modifications in 
test content and in the test administration procedure. With a better 
understanding of the test formats and procedures we pilot tested, we were able 
to generate two forms of the test that we felt were comparable in content. In 
the next stage of pilot testing we expected to be able to determine if they were 
also comparable statistically, 
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The August pilot testing reinforced the need for rigorous training of test 
administrators and led to the development of a workshop for test 
administrators that was conducted prior to the October pilot testing. 



Speaking Test: Establishing Equivalence of Forms 

On the basis of the August pilot test results, two forms of a final 
speaking test were assembled for equivalence testing during October 1992, The 
changes discussed above were incorporated in the two forms. 

Each form of the test included an unscored, 11-item pretest (see Butler et 
al., 1991, p, 15, for a discussion of the pretest) and the two sections that had 
been under development as prototypes: (a) the structured section with the 
sentence repetition task and (b) the picture/story section with structured 
questions about the story. Each test form had a total of 30 scored items. The 
structured section had 20 items; the picture/story section had 10. 

Table 3 shows the feature distribution across Sections 2 (structured 
section) and 3 (picture/story section) for both forms of the test for the October 
1992 pilot testing effort. As the row totals for both forms indicate, each feature 
was tested at least twice on each form and most were tested three or four 
times. Overall feature representation on the two test forms was parallel 
though not identical. 
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Table 3 

Feature Distribution Across Sections and Forms for the Speaking Test, October 1992 Pilot 







Form A 






Form B 




Feature 


Section 2^ 


Section 3^ 


Total 


Section 2 a 


Section 3 b 


Total 


Plural 


2 


2 


4 


2 


2 


4 


Demonstrative 


2 


0 


2 


2 


0 


2 


Third Person 
Singular 


2 


1 


3 


2 


1 


3 


Reflexive 


2 


0 


2 


2 


0 


2 


Past "was/were' 


2 


2 


4 


2 


1 


3 


Present "is/are" 


2 


1 


3 


2 


1 


3 


Possessive 


2 


2 


4 


2 


2 


4 


Wh-locative 


2 


1 


3 


2 


1 


3 


Negation 


2 


1 


3 


2 


1 


3 


Distributive "be" 


2 


0 


2 


2 


1 


3 


Column Total 


20 


10 


30 


20 


10 


30 



Structured section, sentence repetition task. 



^ Picture/story section, structured questions about story. 

October 1992 Pilot Testing • 

The October pilot testing took place in three schools in the district, 52nd 
Street School, Manhattan Place School, and Normandie Avenue School, at 
Grades K, 2, 4, 5, and 6. Table 4 provides the distribution of students across 
schools, grade levels, and forms of the test. 
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Table 4 



Pilot 


Grade 


K 




2 


4 






5 6 


School Form^ 


A B 


A 


B 


A 


B 


A 


BAB 


School 1 

(52nd St. School) 


6 7 


6 


5 


5 


4 




- 6 5 


School 2 


6 4 


6 


6 


4 


4 


1 


4 — — 



(Manhattan 
Place) 

Schools 4 13232 — — 14 

(Nomandie Ave,) 



16 12 15 13 -12 10 1 4 7 9 

Note . There were two raters at Schools 1 and 2 and one rater at School 3. - = no students tested. 
^ Form A: N = 51 (across grade levels). Form B: M = 48 (across grade levels). 



The tests were administered by five retired teachers employed as 
substitutes by the district. These administrators received one day of training at 
UCLA's Center for the Study of Evaluation. (See Appendix F for a summary of 
the training procedures and Appendix G for the Directions for Test 
Administration.) The training was conducted by the test development staff 
responsible for the project. All five teacher-administrators were enthusiastic 
about the project, the rationale behind it, and their involvement in the process. 

Pilot Testing Results 

In addition to testing for equivalence of forms during this final phase of 
pilot testing before the validation study, interrater reliability was examined, 
and weak items were identified and revised. Each of these issues is diccussed 
in turn below. 
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In the discussions that follow, results are reported for two groups of 
raters: (a) the five retired LAUSD teachers who administered the tests in the 
schools and scored them on the spot, and (b) two members of the UCLA test 
development staff who listened to a subset of the recorded tests and provided 
independent ratings so that interrater reliability could be calculated. For the 
first group, 51 ratings were available for Form A and 48 ratings for Form B. 
For the second group, 17 Form A tests and 14 Form B tests were rescored. 

Equivalence of forms . In order to use the two forms of the speaking test 
as pre and post measures, the equivalence of the two forrr^ must be 
established. That is, we must be able to say with confidence that the two forms 
are measuring the same thing (content equivalence) and are doing so in the 
same way (statistical equivalence). 

In developing story lines and writing test items, we attempted to assure 
that the language difficulty and the interest level of the test content were 
appropriate for K-6 students. Feedback from teachers and test administrators 
guided our revisions to help assure the appropriateness of the content for the 
target population. In addition, the features being tested were distributed in a 
balanced way across test sections and test forms to help assure for equivalence 
of feature coverage. 

To determine statistical equivalence of the two forms of the speaking 
test. Form A and Form B, an analysis of variance (ANOVA) was run on the 
test results separately for both the LAUSD raters and the UCLA raters. 
Table 5 provides the results for the LAUSD raters. 
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Table 5 

Analysis of Variance Results for the LAUSD Raters for Form A and Form B of the Speaking 
T^.qt. Qctohpr 1992 Pilot 



Sum of Mean 
Source df Squares Squares F Ratio F Prob. 



Between 1 43.8788 43.8788 1.0189 .3153 
Groups 

Within 97 4177.2929 43.0649 
Groups 

Total 98 4221.1717 



Note. Raterli = 5. 



An F ratio of 1.02 (p. < .05) indicates that there is no significant difference in 
forms for this group of raters. Table 6 provides the results for the UCLA 
raters. 



Table 6 

Analysis of Variance Results for the UCLA Raters • . Form A and Form B of the Speaking 
Test, October 1992 PilQt 



Source 


df 


Sum of 
Squares 


Mean 
Squares 


E Ratio 


F Prob. 


Between 
Groups 


1 


21.9035 


21.9035 


.5615 


.4597 


Within 
Groups 


29 


1131.1933 


39.0067 






Total 


30 


1153.0968 









Note . Rater N = 2. 



An F ratio of .56 (p. < ,05) indicates that for the UCLA raters there is also no 
significant difference in the two forms of the test. The results of the P,M. 
Correlation analysis (r = ,93) between Form A and Form B along with the 
ANOVA results indicate that Form A and Form B of the speaking test are 
parallel forms. 




Interrater reliability . The interrater reliability was high for both forms 
of the tests for the October 1992 nilot testing. The interrater reliability for Form 
A with 17 speaking tests was .917 and for Form B with 14 speaking tests was 
.932. 

Though the sample size was limited, these results show that the group 
of LAUSD raters and the pair of UCLA raters were consistent in their 
evaluations of the speaking samples across both forms of the test. These 
findings are encouraging and suggest that the 1-day rater training 
successfully prepared the teachers to administer the test and score the student 
responses with a high degree of accuracy. 

Revision of weak items . Item statistics (item mean, item standard 
deviation, item total correlation, and "reliability of scale when item removed'* 
were used to help us identify weak items (items with low item-total 
correlations) on both forms of the speaking test. Three items requiring further 
scrutiny were identified on Form A and 10 items on Form B. The test 
developers reexamined the content of those items and made modifications. 
The revised items will be examined again during the 1993 validation phase. 

Discussion 

The results of the October 1992 pilot testing indicate that two parallel 
forms of the speaking test have been developed for use with K-6 grade students 
in the LAUSD LDPAAS. The test, which focuses on 10 feature differences 
between MAE and AAL, must now undergo validation procedures. 
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New Writing Test Prompts 



During the 1st year of test development, two types of writing prompts 
were pilot tested — topics with pictures and topics without pictur'es. While both 
types of prompts elicited ratable samples from the students, the topics w?th 
pictures generated richer language samples. For this reason, the 
recommendation was made to continue developing writing prompts that 
involved the use of pictures (Butler et aL, 1991, p. 9). 

The writing prompts piloted in Year 1 were free production or open- 
ended prompts in that the students were asked to write a story about a given 
topic. There was no attempt to control the language they produced. In 
general, the children at all the grades (3-6) tested were able to write stories in 
response to the prompts, though there were differences in performance across 
grades with the older students producing more language. 

From a scoring perspective, the open-ended stories proved problematic 
because there was no way to assure the production of the features we wanted to 
assess. Some topics lend themselves to the use of some features more than 
others, but there is no way to confidently predict the use of specific features. 

During the July 1992 meeting with Lorraine Cole, pictures for writing 
prompts and the scoring scheme were discussed at length. As a result of those 
and additional discussions, test development staff decided to modify the format 
of the writing test. Picture prompts would still be used, but a short story would 
be developed to accompany each picture, and specific questions would be asked 
about the picture and story. Those questions would be framed to elicit the same 
10 features being assessed in the speaking test. We felt that this modification 
would allow for more efficient use of testing and scoring time and would 
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provide information about students' use of MAE in writing that would be 
parallel to the information about MAE use in speaking. 

Focus Groups 

To assist in selecting pictures for writing prompts that would appeal to 
the target population, UCLA test developers conducted focus groups on October 
8, 1992, at 99th Street School with 3rd graders (6 girls, 6 boys) and 6th graders 
(7 girls, 5 boys). They were shown 11 pictures, one at a time, which had been 
grouped into three categories: people, animals, and fantasy. The children 
were asked if they liked each picture and what kind of stories they could think 
of about it. In four instances, a single page had a pair of pictures that were 
thematically linked. Those four pairs are counted as single pictures. The 
focus group pictures are provided in Appendix H and are lettered A-K. 

The pictures of Mickey Mouse (J), the cat (H), and the two dogs (I) 
appealed the most to the 3rd graders. Topics suggested by the 3rd graders 
included: (a) why Mickey is dressed/standing the way he is in the picture, 
(b) why the cat is scared/hiding, and (c) the relationship between the two dogs. 
A topic suggested by the 3rd graders that was not related to any of the pictures 
was dancing. 

The 6th graders liked the following pictures the most: the girl looking in 
the mirror (C), the boy looking up (D), the cat (H), the two dogs (I), and Mickey 
Mouse (J). They suggested adding pictures of famous people such as Michael 
Jordan, Magic Johnson, and Larry Bird. They felt sports in general would 
provide good topics. They also thought pictures of dinosaurs would be 
interesting to write about. The 6th graders indicated that they would have no 
trouble coming up with ideas for any of the pictures. 
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Prompts 

Eight writing prompts have been developed and will be pilot tcjted 
during winter 1993. Each prompt includes a picture, a short story based on the 
picture, and a series of four to six questions that attempt to elicit answers 
requiring the same features we are assessing with the speaking test. Two 
answer formats will be pilot tested. One format allows the student to freely 
generate an answer with the hope that the question is structured enough to 
forv;e a specific response. The second format asks the same questions but 
frames each response by actually providing a single-sentence answer with a 
blank for the word or words which contain the feature being tested. 

Pilot testing will provide the information needed to make a decision 
about the format to be used and the number of pictures and items to include on 
the final writing test forms. The eight pictures selected for the new prompts 
were chosen from the focus group favorites and from additional pictures 
w^hich reflected suggestions made by the students in the focus groups. 

Discussion 

The change in format of the writing test has meant a shift from a 2-day 
process approach involving a free writing sample (Butler et al., 1991) to a 
structured approach that involves sentence-level written answers to specific, 
focused questions. The format change allows us to concentrate our 
assessment on a set of specific feature differences between MAE and AAL. If 
we are successful in eliciting the desired features in the written answers to the 
questions, we will have assessment data in the written modality that is more 
comparable to the speaking test data than were data from the free writing 
samples, which often did not produce the features of interest in either MAE or 
AAL. 

ii'i 
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Summary 



Work during Year 2 of the test development project described here has 
produced two parallel forms of a speaking instrument to help the LAUSD 
assess the effects of its LDPAAS on student proficiency in MAE. To this end, 
prototype formats were pilot tested and revised, and then two forms of the 
instrument were pilot tested to assure comparability of forms. 

In addition to finalizing the speaking instrument, eight new writing 
prompts based on pictures and short stories were developed and will be pilot 
tested during winter 1993 as part of the third-year tasks for the project. The 
pilot testing results will allow us to finalize two forms of the writing test and 
move to the validation phase for both instruments. 
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Appendix A 

Examples of the 10 Features of Difference 
Between MAE and AAL Being Tested 
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The examples below each give (a) the name of the grammatical feature 
being tested, (b) a question or prompt which should elicit the feature as part of 
the response, and (c) possible responses in MAE and AAL, 

1, plural 

How many buildings do you see? (A plural answer with number is desired,) 

MAE: Two buildings, 
AAL: Two building . 



2. demonstrative 

Which buildings are almost as tall as Calvin? 

MAE: Those buildings. 
AAL: Them buildings. 



3. third person singular 

After Calvin starts to grow bigger, tell me what he does next. He 

MAE: He crawls/goes , . . 
AAL: He crawl/go . . , 

4. reflexive 

Who do you think put them there? 

MAE: They got there themselves . 
AAL: They got there theyself . 

5. past "was/were" 

What was happening before the lights went out? 

MAE: They were watching/sitting. . . 
AAL: They was watching/sitting. . . 
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6. present "is/are" 

What did I say they're doing? 



MAE: They are/TheyVe watching/sitting . 
AAL: Thev watching/sitting . . . 



7. possessive 

Can you tell me where the cat is sitting? 

MAE: On Jon!s lap. 
AAL: On Jon lap. 



8. wh-locative 

What does he ask Garfield? 

MAE: Do you know where the candles are? 

Where are the candles ? 
AAL: Do you know where the candles are at? 

Where are the candles at? 



9. negation 

What can they see in the dark? 

MAE: Thev can't see anything , 
AAL: They can't see nothing . 



10. distributive "be" 



What does Garfield always do? 



MAE: He is always eating, 
AAL : He always be eating. 
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Directions for Test Administrators 



Speaking Test 



August 1992 Pilot 
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Directions for Test Administrators: Speaking Test 

August 1992 



Test Sections 

The speaking tests that you will be administering to your students 
consist of two sections. The first section contains two cartoons with questions 
about each cartoon. The second section contains two stories with questions 
about each story. Pictures accompany each story. Each question focuses on 
one feature of difference between Mainstream American English (MAE) and 
African American Language (AAL). 

Test Recording 

All tests must be tape recorded so that we can listen to the student 
responses and determine how we might modify the test content to more 
effectively assess the language ability of the students. You will be given two 
cassette tapes. We,have indicated in the test booklet when to start and stop the 
tape recorder. Be sure to record only one student on each side of the tape. Be 
sure you start at the beginning of the side each time. It is important that each 
student give his/her full name at the beginning of the tape. Also, be sure that 
you label each side of the cassette with the appropriate student's name. 

Student Selection 

We ask that you select four students (2 males and 2 females) to take the 
test whom you consider to be the least proficient speakers of MAE in your 
class. In this way we will be better able to determine if the items on the test 
will elicit AAL because the students who are less proficient in MAE will be 
more likely to respond in AAL than would the more proficient MAE speakers. 

Student Answers 

The directions ask the student to answer questions using MAE. For 
some questions it is absolutely critical that the student answer in a complete 
sentence. In those instances the directions tell you how to prompt the student 
to provide a more complete answer. In other instances you should be able to 
judge whether or not the feature is present without hearing a complete 
sentence. 

Student Information 

Provide the information requested on the ft'ont of each test booklet before 
you begin the test. 
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Scoring 



There is a place right in the test booklet for you to score each response. 
You will indicate whether the response given by the student was in MAE, 
AAL, or whether the Feature was simply Not Present in the response (FNP). 
There are boxes where you will check either MAE, AAL, or FNP. 

We have provided examples of what the students might say in both MAE 
and AAL to help guide your decisions about the answers the students give. In 
every instance, you should focus on what is underlined in the examples as you 
evaluate a response. It is important to remember, however, that the students 
may give other equally appropriate answers that contain the feature being 
tested. If you are unsure about a given answer, write down in the test booklet 
what the student said and do not check any of the boxes. 

There are several questions throughout the test that ask "How many" of 
something the student sees in a picture. What we are testing for with those 
questions is the student's ability to provide the plural "s" on the noun. For 
those items, the example answers we have provided say "X," whatever the 
noun is. The reason for this is that the exact number the student gives is not 
important. If the student gives the wrong number but provides the plural "s" 
on the noun, you should put a check in the MAE box. 



Sample Question: 

Say: 'How many turtles do you see in this picture? " 

Student's answer: "Three turtles/* 



Responses: 


PLURAL ^ 






MAE 


AAL 






X turtles 


X turtle 










FNP 


17. PLURAL 









(Note: A plural answer WITH A NUMBER is desired. If the student responds with just a 
number ask: "X what?") 



Upon completion of the test, total the number of MAE responses and record the 
total in the box provided on the cover sheet. 

l'J2 
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Appendix C 



Feedback Questionnaire 



Spealdi^Test 



August 1992 Pilot 
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UCLA test developers: 

Frances Butler, Edynn Sato, Joe Plummer 



August 1992 



Questionnaire 



Thank you very much for agreeing to give the speaking test we are 
developing for the Language Development Program for African American 
Students to a few of your students. We are extremely interested in feedback 
from you regarding the test administration procedure and your sense of how 
well the students responded to the tasks. To this end, we are providing a short 
list of questions below that we would very much appreciate your taking the 
time to answer. Your answers will help us improve the testing instrument. If 
you have any additional comments, please call us at (310) 206-1532. We would 
welcome your input. 



1, Did you find the two formats, (1) cartoon with questions and (2) story 
with questions, easy to administer? How could the administration be 
made easier? 



2. Did you understand the category "Feature Not Present (FNP)" for 
evaluating student answers? If not, what seemed confusing? 



3. Can you suggest alternative questions for eliciting the features we are 
attempting to analyze? If so, could you provide specific examples? 



4. (a) In general, how did you like the cartoons and story pictures? 

(b) Did the students seem to respond well to them? 

(c) Do you have comments about specific pictures? 

(d) Did the students have any comments we should be aware of? 

(e) Were any of the pictures confusing to the children or somehow 
inappropriate? If so, please explain. 



(Note: Please use the back of this sheet to expand your comments.) 
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Appendix D 
Scoring Form 



Speaking Test 



August 1992 Pilot 
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August '92 Pilot 

LAUSD MAE Project 
Scoring Form 

Tape Number Student I.D. # 

(School, Grade, Student) 

Side A B 



Item 
# 


Feature being 
Sought 


Notes 

Teacher 
Correct? 


1 


PRESENT "IS/ARE" 


1 1 

□ 


2 


POSSESSIVE 


\ 1 

□ 


3 


PLURAL 


1 ! 

□ 


4 


POSSESSIVE 


1 1 

□ 


5 


NEGATION 


1 ! 

□ 


6 


THIRD PERSON SINGULAR 


1 1 

□ 


7 


WH-LOCATTVE 


1 — 1 


8 


WH-LOCATIVE 


1 1 

□ 


9 


REFLEXIVE 


1 1 

□ 


10 


PAST "WAS/WERE" 


1 1 

□ 


11 


NEGATION 


[ 1 

□ 


12 


PRESENT "IS/ARE" 


1 1 

□ 


13 


THIRD PERSON SINGULAR 


1 1 

□ 


14 


THlKD P??RSON SINGULAR 


LJ 


15 


POSSESSIVE 


□ 


16 


WH-LOCATIVE 


□ 


17 


PLURAL 


□ 


18 


DEMONSTRATIVE 


□ 


19 


DISTRIBUTIVE ASPECT 


□ 
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20 


DEMONSTRATIVE 


1 1 

□ 


21 


POSSESSIVE 


1 1 


22 


PLURAL 


1 1 


23 


PLURAL 


1 1 

□ 


24 


PAST "WAS/WERE" 


1 1 

□ 


25 


THIRD PERSON SINGULAR 


1 1 

□ 


26 


PRESENT WASAVERE 


1 1 

□ 


27 


POSSESSIVE 


1 1 

□ 


28 


REFLEXIVE 


1 1 

□ 


29 


THIRD PERSON SINGULAR 


I 1 


30 


NEGATION 


1 1 

□ 


31 


PAST WAS/WERE 


1 1 

□ 


32 


PLURAL 


I 1 

□ 


33 


PLURAL 


1 1 


34 


POSSESSIVE 


1 1 


35 


REFLEXIVE 


I — 1 


36 


POSSESSIVE 


1 1 


37 


PAST WAS/WERE 


I 1 


38 


DEMONSTRATIVE 


□ 


Total Number of 
items marked as 
correct by teacher 





I?7 
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Appendix E 



Response Sheets for Part III: 
Story Section for Form A and Form B 



Speaking Test 



October 1992 Pilot 
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Part Id: Story Section 



Directions: Listen for the student's use of Mainstream American English 
(MAE). If the student uses MAE for the feature tested, put a check in the box 
marked MAE on the scoring sheet, If the student uses African American 
Language (AAL), put a check in the box marked AAL on the scoring sheet. If 
the student provides a response that is not being tested for, put a check in the 
box marked FNP (Feature Not Present). Examples of MAE and AAL 
responses are provided along with the prompt on the back of the picture. 

Say: ^T^ow I am going to tell you a story. I will show you some pictures that go 
along with it and ask you some questions about the story. When you answer 
the questions, I want you to use Mainstream American English* Do you have 
any questions?^ 

Go to the back of Picture 1. Record answers from the Story Section here. 



MAE AAL FNP 



21. THIRD PERSON 
SINGULAR 








22. WH-LOCATIVE 








23. NEGATION 








24. PRESENT 
"IS/ARE" 








25. PAST 

nVASAVERE" 








26. PLURAL 








27. POSSESSIVE 








28. POSSESSIVE 








29. PAST 

"WASAVERE" 








30. PLURAL 









Total Number of 
MAE Responses 
in Part III: 
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Form A 



Partni: Story Section 



Directions: Listen for the student^s use of Mainstream American English 
(MAE). If the student uses MAE for the feature tested, put a check in the box 
marked MAE on the scoring sheet. If the student uses African American 
Language (AAL), put a check in the box marked AAL on the scoring sheet. If 
the student provides a response that is not being tested for, put a check in the 
box marked FNP (Feature Not Present). Examples of MAE and AAL 
responses are provided along with the prompt on the back of the picture. 

Say: ^*Now I am going to tell you a story. I will show you some pictures that go 
along with it and ask you some questions about the story. When you answer 
the questions, I want you to use Mainstream American English* Do you have 
any questions?" 

Go to the back of Picture i. Record answers from the Story Section here. 



MAE AAL FNP 



21. DISTRIBUTIVE 
''BE*' 








22. POSSESSIVE 








23. PLURAL 








24. PAST 

"WAS/WrRE" 








25. PLURAL 








26. THIRD PERSON 
SINGULAR 








23. PRESENT 
"IS/ARE" 








28, POSSESSIVE 








29. NEGATION 








30. WH-LOCATIVE 






1 



Total Number of 
MAE Responses 
in Part III: 
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Form B 



Appendix F 



Summary of Training Procedure 
for Test Administrators 



Speaking Test 



October 1992 Pilot 
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Five retired teachers employed as substitutes by the Los Angeles Unified 
School District (LAUSD) received 1 day of training (5 hours) at UCLA's Center 
for the Study of Evaluation to prepare them to administer the two forms of the 
speaking test for the October 1992 pilot. Training Vv^as conducted by three 
researchers involved in the development of the test. 

Test administrators were informed of the purpose of the LAUSD African 
American Language Development Project, its history, and their role in the test 
development process. They were then given general directions for test 
administration which included an overview of the three main sections of the 
test (i.e., pretest, sentence repetition section, picture/stoiy section), guidelines 
and examples for standardizing test administration (e.g., how to elicit 
students' responses), instructions for recording and scoring student 
responses, and directions for completing a test administration log. Procedures 
for tape recording student responses were also reviewed. 

Test administrators were then given an item-by-item review of each 
form of the speaking test. Examples of student responses were provided for 
each item with guidance on how they should be scored. Administrators were 
reminded to use natural language to elicit student responses and to prompt 
students according to the script provided for each item. 

Following the review of the directions for administration and the tasks, 
test administrators practiced administering and scoring the two test forms. 
First, two UCLA trainers role-played the parts of a test administrator and a 
student while the administrators-in-training listened and scored the 
"student's" responses. Test administrators were then given time to practice 
administering both forms of the assessment to a partner while trainers 
observed and commented on their techniques. 
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At the end of training, school assignments were reviewed and the 
design for test administration/data collection explained. 
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Directions for Test Administration 



Speaking Test 



October 1992 Pilot 
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Directions for Test Administration 
Spealdng 

LAUSD Language Development Program 
for African American Students 
October 1992 Pilot Test 

Test Sections 

The speaking test that you will administer to students consists of three 
sections: 

Parti: A short pretest, which involves (a) word repetition and 
(b) short answers to questions about pictures 

Part II: A sentence repetition section 

Part III: A story section, which asks students to answer questions 
about a story you tell them. Pictures accompany the story. 

The test focuses on features of difference between Mainstream American 
English (MAE) and African American Language (AAL). 



Part I: Fittest 

There are two sections in this pretest. In the first section, you will ask the 
student to repeat a list of nine words. You should say each word only once. If 
the student produces the underlined sound in each word, put a check (V) on. the 
line next to the word. If the student does not produce the sound, put a zero (0) 
on the line next to the word. In the second section, you will show the student 
two pictures one at a time, and ask the student to answer a question about each 
picture. If the student provides the desired response, no mark should be made 
in the test booklet. If the student does not provide the desired response, record 
the student's response on the line provided in the test booklet, 

[Note: Even if the student is unable to produce any of the appropriate sounds 
during the pretest, proceed to Parts II and III.] 



Part 11: Sentence Repetition Section 

You will read a series of sentences to the student one at a time, and the student 
will repeat each sentence. You should read each sentence to the student only 
once. You should read each sentence at a normal rate of speech with no 
special emphasis on the underlined sound(s) and word(s). Listen for the 
student's use of MAE. If the student produces the underlined feature, put a 
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check i^l) in the box next to the sentence. If the student does not produce the 
feature, put a zero (0) in the box next to the sentence. After you have finished 
administering the whole test, total the number of checks in Part II and enter 
the total on the appropriate line on page 4 of the test booklet. 



Part DL Stoiy Section 

You will read a story to the student and ask the student questions about the 
story and the pictures accompanying the story. Listen for the student's use of 
MAE. If the student uses MAE for the feature tested, put a check (^) in the box 
marked MAE on the scoring sheet. If the student uses AAL, put a check in the 
box marked AAL on the scoring sheet. If the student provides a response that 
(a) is neither MAE or AAL or (b) is an AAL form not being sought, put a check 
in box marked FNP (Feature Not Presenc). Examples of MAE and AAL 
responses are provided for each question along with the prompu on the back of 
the picture. After you have finished administering the whole test, total the 
number of checks in Part III and enter the total on the appropriate line on 
page 5 of the test booklet. 

Procedure 

Before you begin, be sure to provide the information requested on the front of 
the test booklet (i.e., date, student's name, teacher's name, student's grade in 
school, school's name, and your name). 

All tests must be tape recorded so that we can listen to the student responses 
and determine how we might modify the test to more effectively assess the 
language ability of the students. You will be given 11 cassette tapes. Be sure to 
record only one student on each side of the tape. Be sure you start at the 
beginning of the side each time. It is important that each student give his/her 
full name at the beginning of the tape. Also, be sure that you label each side of 
the cassette with the appropriate student's name. 

When you and the student are ready, start the cassette recorder. 

Ask the student : 

"What is your name?" [Note: Be sure the student provides his/her first 
and last name.] 

Go to the test booklet and begin the procedure. 

When the student has answered the last question in Part III, stop the 
recorder. 

Total the number of MAE responses in Parts II (p. 4) and III (p. 5) and write 
the combined total in the box on the front of the test booklet. 
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Provide the information asked for on the Test Administration Log: 

(1) Testing location (e.g., library, teachers' lounge, classroom) 

(2) Conditions (e.g., noise level, number of other people in the room, 
interruptions) 

(3) Number of students tested at the location 

Please use this sheet to provide any comments you have about the testing 
process and/or the test content. Your comments will help us refine this 
speaking test. 

Comments: 
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Appendix H 



Focus Group Pictures 
October 1992 



This appendix includes the 11 pictures used in focus groups with 3rd 
graders and 6th graders at 99th Street School on October 8, 1992. The pictures 
were grouped into three categories: people (A-F), animals (G-I), and fantasy 
(J"K). 
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